Skip to content

gh-79638: Treat an unreachable robots.txt as "disallow all" (GH-138555)#149541

Closed
serhiy-storchaka wants to merge 1 commit intopython:mainfrom
serhiy-storchaka:robotparser-unreachable2
Closed

gh-79638: Treat an unreachable robots.txt as "disallow all" (GH-138555)#149541
serhiy-storchaka wants to merge 1 commit intopython:mainfrom
serhiy-storchaka:robotparser-unreachable2

Conversation

@serhiy-storchaka
Copy link
Copy Markdown
Member

@serhiy-storchaka serhiy-storchaka commented May 8, 2026

Disallow all access in urllib.robotparser if the robots.txt file is unreachable due to server or network errors.

…thonGH-138555)

Disallow all access in urllib.robotparser if the robots.txt file is
unreachable due to server or network errors.
@serhiy-storchaka
Copy link
Copy Markdown
Member Author

I am puzzled. Tests on Android and iOS fail because they modify urllib.requests._opener. But there is an explicit cleanup code for it. The only difference is that it is now a class cleanup instead of a separate test cleanup, and the server and the thread are ran at the class level (for performance).

@serhiy-storchaka
Copy link
Copy Markdown
Member Author

It turned out that the explicit check is not needed, because the current code passes the required tests. It may be needed in future, when we change the code (for example, to allow reloading of robots.txt).

@serhiy-storchaka serhiy-storchaka deleted the robotparser-unreachable2 branch May 8, 2026 20:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

awaiting core review needs backport to 3.13 bugs and security fixes needs backport to 3.14 bugs and security fixes needs backport to 3.15 pre-release feature fixes, bugs and security fixes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant